Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Customize shortcut modifier and forward Ctrl and Shift to the device #1598

Merged
merged 17 commits into from
Aug 1, 2020

Conversation

rom1v
Copy link
Collaborator

@rom1v rom1v commented Jul 16, 2020

This PR replaces the two previous attempts #1446 and #1465.

It forwards Ctrl and Shift to the device. This allows for example to press Ctrl+t in Firefox to open a new tab, Ctrl+w to close the current tab. Ctrl+c or Ctrl+d can now be injected in Termux.

This also greatly simplifies text selection, by using the keyboard (Shift+arrows, Ctrl+a…), and simplifies copy-paste (Ctrl+c) and cut-paste (Ctrl+x). Ctrl+z could be used to cancel an action, like a change in a text field for example.

As a consequence, scrcpy shortcuts are not bound to Ctrl anymore, but to (left) Alt and (left) Cmd by default. This modifier can be changed by a command-line parameter (--shortcut-mod), as explained here.

Here is the list of the changes:

  • all shortcuts now use the specified modifier key (called MOD), on all platforms
  • Ctrl and Shift are forwarded to the device
  • to copy or cut a text, Ctrl+c and Ctrl+x typically work, because they are forwarded to the device, which, in a text field, copies the text into the clipboard… and the clipboard is now automatically (since v1.14) synchronized with the computer keyboard
  • pressing Ctrl+v synchronizes the computer clipboard to the device before sending the key event, therefore pasting typically works seamlessly
  • seamless copy-paste now also works before Android 7 with Ctrl keys
  • MOD+c, MOD+x and MOD+v are also provided to inject COPY, CUT and PASTE keycodes (see README for details)
  • MOD+Shift+v injects the computer clipboard content as a sequence of key events (this is still useful sometimes)
  • to be able to bind CUT to MOD+x, the "resize to fit" shortcut (which removes the black borders) has been changed to MOD+w

This fixes #555 and #942.

Feedbacks (and reviews) welcome :)

Binaries

EDIT: just take v1.15 release.

old For windows users, take both `scrcpy.exe` and `scrcpy-server`, and replace them in your scrcpy v1.14 release.

For other platforms, take scrcpy-server and build only the client.

  • scrcpy.exe
    SHA256: ceab026bff28c8851456e30d3e1449ad25ef6aedeec6fcac00ad508e2f67a9cf
  • scrcpy-server
    SHA256: b7b4f2f5d008cf6191c8fab6f74cc43bfc78bd33c4a4100274a9f4fa717b59e8

rom1v added 2 commits July 17, 2020 00:00
When the client requests to set the clipboard, it may request to press
the PASTE key in addition. To be a bit generic, it was stored as a flag
in ControlMessage.java.

But flags suggest that it represents a bitwise union. Use a simple
boolean instead.
The condition "cmd" was always before "shift" in all expressions except
4.
@rom1v
Copy link
Collaborator Author

rom1v commented Jul 16, 2020

Oh, when pressing Alt+c, SDL generates a text event (SDL_TextInputEvent) containing c (as if Alt was not pressed).

As a consequence, when --prefer-text is passed, a c is inserted on such a shortcut 😕

EDIT: fixed by 19691d6, I updated the binaries.

@Helaer
Copy link

Helaer commented Jul 17, 2020

Hello, under the default ALT shortcut, ALT+W has no effect on removing black borders.

@brunoais
Copy link
Contributor

brunoais commented Jul 17, 2020

@Helaer It works fine for me on a PT-PT keyboard. What's your keyboard layout?

@rom1v Works awesome to me. After some use, I settled with --shortcut-mod="ralt" for me (AltGr). It's much easier and faster to stick out the thumb without that much deflection. I do understand well that there are keyboards without such key. Being able to choose is great.
I tried around to use app shortcuts and they worked well as expected. 15 mins using the phone with multiple Ctrl shortcuts and such didn't show any signs of issues or bugs for me. Including using AltGr+W to resize the window.
Btw, In my linux system, I did a full build with the source code instead of taking your prebuilt server. For some reason, I was getting a segmentation fault and I was unable to fix it quick.
So I did a full build.

@Helaer
Copy link

Helaer commented Jul 17, 2020

@Helaer在PT-PT键盘上对我来说效果很好。您的键盘布局是什么?

@ rom1v对我来说很棒。使用一段时间后,我--shortcut-mod="ralt"为我定居了(AltGr)。伸出拇指而没有那么大的挠曲会更加轻松快捷。我非常了解有些键盘没有这样的键。能够选择很棒。
我尝试使用应用程序快捷方式,它们按预期运行良好。15分钟使用带有多个Ctrl快捷方式的电话,这对我来说没有任何问题或错误的迹象。包括AltGr+W用于调整窗口大小。
顺便说一句,在我的Linux系统中,我使用源代码进行了完整构建,而不是使用您的预构建服务器。出于某种原因,我收到了segmentation fault,但无法快速修复。
所以我做了一个完整的构建。

I didn't pass in a set of bytes, the default shortcut used, ALT+W can not remove the black border of the window.

@rom1v
Copy link
Collaborator Author

rom1v commented Jul 17, 2020

@Helaer

Hello, under the default ALT shortcut, ALT+W has no effect on removing black borders.

Which platform/window manager? On Debian+XFCE, Alt+w works fine.
What about Cmd+w on your computer (Cmd is the Windows key on some keyboards)?

@brunoais Thank you for your feedbacks 👍

After some use, I settled with --shortcut-mod="ralt" for me (AltGr).

On a French keyboard, AltGr is used to generate many chars (AltGr+0 is @, AltGr+6 is |, and many many others).

I did a full build with the source code instead of taking your prebuilt server. For some reason, I was getting a segmentation fault and I was unable to fix it quick.

Oh, a segmentation fault on the client side or device side? (does it appear in adb logcat)
Can you reproduce?

Add -Db_sanitize=address to your meson command, and retry. It will give more details if it's on the client side.
For example, you can update your meson configuration and rebuild:

meson configure x -Db_sanitize
ninja -Cx
./run x

@Helaer
Copy link

Helaer commented Jul 17, 2020

@海拉尔

您好,在默认的ALT快捷方式下,ALT + W对删除黑色边框无效。

哪个平台/窗口管理器?在Debian + XFCE上,Alt+ w可以正常工作。
什么Cmd+ w您的计算机上(Cmd是Windows某些键盘上的键)?

My system is Windows 10.

@rom1v
Copy link
Collaborator Author

rom1v commented Jul 17, 2020

I just tested on Windows 10: I resized the scrcpy window so that there are black borders, then I pressed Alt+w, it works for me, the window is automatically resized so that black borders are gone.

@Helaer
Copy link

Helaer commented Jul 17, 2020

我刚刚在Windows 10上进行了测试:我调整了scrcpy窗口的大小以使其具有黑色边框,然后按Alt+ w,它对我有用,该窗口会自动调整大小以使黑色边框消失。

I restarted my computer and it responded, guessing that other applications blocked the key combination.

@rom1v rom1v mentioned this pull request Jul 17, 2020
2 tasks
@brunoais
Copy link
Contributor

brunoais commented Jul 17, 2020

@Helaer

Hello, under the default ALT shortcut, ALT+W has no effect on removing black borders.

Which platform/window manager? On Debian+XFCE, Alt+w works fine.
What about Cmd+w on your computer (Cmd is the Windows key on some keyboards)?

@brunoais Thank you for your feedbacks +1

After some use, I settled with --shortcut-mod="ralt" for me (AltGr).

On a French keyboard, AltGr is used to generate many chars (AltGr+0 is @, AltGr+6 is |, and many many others).

I did a full build with the source code instead of taking your prebuilt server. For some reason, I was getting a segmentation fault and I was unable to fix it quick.

Oh, a segmentation fault on the client side or device side? (does it appear in adb logcat)
Can you reproduce?

Add -Db_sanitize=address to your meson command, and retry. It will give more details if it's on the client side.
For example, you can update your meson configuration and rebuild:

meson configure x -Db_sanitize
ninja -Cx
./run x

@Helaer

Hello, under the default ALT shortcut, ALT+W has no effect on removing black borders.

Which platform/window manager? On Debian+XFCE, Alt+w works fine.
What about Cmd+w on your computer (Cmd is the Windows key on some keyboards)?

@brunoais Thank you for your feedbacks +1

After some use, I settled with --shortcut-mod="ralt" for me (AltGr).

On a French keyboard, AltGr is used to generate many chars (AltGr+0 is @, AltGr+6 is |, and many many others).

I have such too. Except it never worked in scrcpy. I end up using the phone's virtual keyboard for those characters.
When that's solved, then I may change the combination to --shortcut-mod="ralt+rctrl", for example.
The phone is running Android 6.0.0

Oh, a segmentation fault on the client side or device side? (does it appear in adb logcat)

I'm not sure, I can only see a connection lost on logcat. No process dump or stack trace.

Can you reproduce?

When I forcefully replace the server code you uploaded there into the resulted client compilation, yes.

Add -Db_sanitize=address to your meson command, and retry. It will give more details if it's on the client side.
For example, you can update your meson configuration and rebuild:

meson configure x -Db_sanitize
ninja -Cx
./run x

When I do it like that, everything runs right. I think ninja is recompiling when I run it like that. So I think I would need to run it in some different way so it uses the server you uploaded there while using the client code from my side.

Regardless, if I checkout cdd6f186, optionally run meson and then run ninja, all runs fine and smoothly. 👍

@rom1v
Copy link
Collaborator Author

rom1v commented Jul 17, 2020

I have such too. Except it never worked in scrcpy. I end up using the phone's virtual keyboard for those characters.

Even with scrcpy --prefer-text?

When I do it like that, everything runs right.

What about:

meson configure x -Dprebuilt_server=/path/to/your/prebuilt/scrcpy-server
...

?

@brunoais
Copy link
Contributor

brunoais commented Jul 17, 2020

I have such too. Except it never worked in scrcpy. I end up using the phone's virtual keyboard for those characters.

Even with scrcpy --prefer-text?

When I do it like that, everything runs right.

Yeah... It does solve... Thank you.
However, I still will lose the ability with other programs that are listening to keys (you exemplify games)... Not really something I'd want...

What about:

meson configure x -Dprebuilt_server=/path/to/your/prebuilt/scrcpy-server
...

?

I didn't know that option existed I'll try that tomorrow.

@brunoais
Copy link
Contributor

What about:

meson configure x -Dprebuilt_server=/path/to/your/prebuilt/scrcpy-server
...

?

I didn't know that option existed I'll try that tomorrow.

No crash if I do like that.

@AzureRaptor
Copy link

AzureRaptor commented Jul 20, 2020

@rom1v Just want to mention that on my platform, Windows 10, Samsung Galaxy S5, the two most recent binaries are working perfectly. Have tested passing ctrl keys for bash in termux, and everything I've thought of has been flawless. I'm using the "hacker's keyboard" keyboard replacement app also, if that matters. Also tested scrcpy builtins for turn off screen, pull menushade, switch programs, etc. All working fine.

Thank you so, so much for this. This is saving me a ton of agony futzing with the soft keyboard in termux.

@rom1v
Copy link
Collaborator Author

rom1v commented Jul 20, 2020

@AzureRaptor Thank you very much for your feedbacks. It's very helpful 👍

@AzureRaptor
Copy link

I have noticed something else that is a bit odd, although not insurmountable thus far.

Somehow, after an indeterminate period of time (it seems to take more than half an hour - I think - I'm sorry I haven't timed this better) scrcpy seems to get into a state where the middle mouse-button no longer works for "home." If I click it, nothing happens. I've given it over a minute to respond during times when the phone is not being used or cpu-stressed.

If I just close and restart scrcpy the behavior stops and middle button works again.

@rom1v
Copy link
Collaborator Author

rom1v commented Jul 21, 2020

Except HOME, everything continued workibg (left-clock, BACK...)?

Did you reproduce it several times?

@AzureRaptor
Copy link

Yes, that's pretty much exactly what I observed. Middle-button failure for HOME was the only unexpected result.

I've observed it at least three times now.

@rom1v
Copy link
Collaborator Author

rom1v commented Jul 22, 2020

If you reproduce it again, could you test whether Ctrl+h (for HOME) still works when middle-clic fails, please?

@rom1v
Copy link
Collaborator Author

rom1v commented Jul 22, 2020

Also, can you reproduce it on a version which do not include the changes from this PR?

@AzureRaptor
Copy link

If you reproduce it again, could you test whether Ctrl+h (for HOME) still works when middle-clic fails, please?

Ok. Have just tested the build from this PR, and ALT-H (which is what I expect for this build, right?) works for home. This is good.

I'll try to test a report on the last stable as soon as I can. I'm glad I kept those binaries around and renamed them. It's basically going to involve me digging out some idle time and letting the phone sit and wait, because I really need the keyboard functionality this build to work sanely at all. 🤣

@AzureRaptor
Copy link

I have an update. Still haven't tested out the older binaries yet, sorry, but just a little while ago this pull got into a state where both middle-click and alt-h did not respond. I'm afraid I closed it before I tested other inputs (was in the middle of working and it was stopping me 😒). Next time this happens, I'll try to see what else is responding at the time. Hopefully this was maybe a one-off fluke.

@rom1v
Copy link
Collaborator Author

rom1v commented Aug 2, 2020

I didn't enable right-Alt by defauly because it is sometimes AltGr, which is used for many characters.

@AzureRaptor
Copy link

AzureRaptor commented Aug 3, 2020

@rom1v I noticed you'd added some commits and merged a couple of days ago. Are the binaries and checksums linked at the top, in the first post, still valid, or are there others somewhere? I'm guessing they aren't and there must be others, but I don't know how to find them. I apologize in advance if this is something very basic to github I should be able to tell from the posts. 😟

@rom1v
Copy link
Collaborator Author

rom1v commented Aug 4, 2020

No, there are no binaries for the current dev branch. I will release v1.15 soon (hopefully this week), which will include these changes.

@AzureRaptor
Copy link

I have some good news and bad news. 😅

On the plus side, I have a logcat file for you. I also can still confirm that all other keystrokes/mouseclicks seem to be working fine. I even enabled both left & right alt as per your cmdline suggestion above; having both works better for me.

On the less-than-plus side, even though I definitely cleared the log as per your instructions further above, the problem didn't show up until I'd been working for several hours. The logcat file is ~3M. I doubt that would paste well here, and it's waaay above even the max size on pastebin. I'm not sure how to get it to you or make it available. I'm open to suggestions.

Thank you very much for your ongoing patience and work on this.

@rom1v
Copy link
Collaborator Author

rom1v commented Aug 6, 2020

the problem didn't show up until I'd been working for several hours.

OK, thank you for the feedback. 👍

I think that the middle-click issue you observe is not related to this PR anyway (and maybe not to scrcpy at all, I can't see a reason why only this key would stop working, you should check whether SDL still receives the event when it fails, by adding a log).

@rom1v
Copy link
Collaborator Author

rom1v commented Aug 7, 2020

I think that the middle-click issue [is] maybe not [related] to scrcpy at all, I can't see a reason why only this key would stop working

@AzureRaptor So I was totally wrong, I failed to find the issue before the release 😞
So now it is fixed in v1.15.1 :D (see #1643).

@AzureRaptor
Copy link

@AzureRaptor So I was totally wrong, I failed to find the issue before the release 😞

Please don't feel bad about this. I'm really sorry I was never able to get you a good logcat, but they just got huge before the problem ever showed up and I couldn't figure out a good way to make them public (aside from something like dropbox or gdrive, which would've been a different kind of irritating).

I really find scrcpy indispensable. I wish I had the cash to support your development; the work you're putting in is amazing.

@elvisisvan
Copy link

elvisisvan commented Mar 6, 2024

i'm on windows 10 scrcpy 2.3.1 (installed using winget) android 13 google pixel 6a and the current default mod keys are LSuper and LAlt is there a way to remove LAlt as mod key? (i want to switch application on phone using alt+tab but instead the focus returns to laptop)

i have tried using --shortcut-mod flag to specifically set only LSuper as mod key but did not work...
scrcpy --otg -s 26021JEGR17923 --shortcut-mod="lsuper"
image
what am i missing here?

@rom1v
Copy link
Collaborator Author

rom1v commented Mar 6, 2024

(i want to switch application using alt+tab but instead the focus returns to laptop)

This is captured by your system so scrcpy is not even aware that you press alt+tab.

@rom1v
Copy link
Collaborator Author

rom1v commented Mar 6, 2024

Anyway, in OTG mode there are no scrcpy shortcuts.

@elvisisvan
Copy link

(i want to switch application using alt+tab but instead the focus returns to laptop)

This is captured by your system so scrcpy is not even aware that you press alt+tab.

if scrcpy is not aware of the alt+tab action why does the mouse and keyboard focus back to laptop?

@rom1v
Copy link
Collaborator Author

rom1v commented Mar 6, 2024

It captures Alt, but if you press Tab in addition, it is captured by the system.

@elvisisvan
Copy link

elvisisvan commented Mar 6, 2024

Anyway, in OTG mode there are no scrcpy shortcuts.

in otg mode when i press either Super or left Alt key the focus moves back to laptop, how can i make it so only when i press Super key the focus will change, not any other key?

@rom1v
Copy link
Collaborator Author

rom1v commented Mar 6, 2024

The mouse capture keys are hardcoded:

static inline bool
sc_screen_otg_is_mouse_capture_key(SDL_Keycode key) {
return key == SDLK_LALT || key == SDLK_LGUI || key == SDLK_RGUI;
}

@elvisisvan
Copy link

elvisisvan commented Mar 6, 2024

The mouse capture keys are hardcoded:

static inline bool
sc_screen_otg_is_mouse_capture_key(SDL_Keycode key) {
return key == SDLK_LALT || key == SDLK_LGUI || key == SDLK_RGUI;
}

what was the last version which it was not hardcoded? why did you hardcode both super key and left alt key, why not just super key?

@rom1v
Copy link
Collaborator Author

rom1v commented Mar 6, 2024

what was the last version which it was not hardcoded?

It is hardcoded since the beginning: 91418c7

why did you hardcode both super key and left alt key, why not just super key?

Because different OS and desktop environments (and virtual machines) capture different keys, so I wanted a working solution for all cases.

@elvisisvan
Copy link

what was the last version which it was not hardcoded?

It is hardcoded since the beginning: 91418c7

why did you hardcode both super key and left alt key, why not just super key?

Because different OS and desktop environments (and virtual machines) capture different keys, so I wanted a working solution for all cases.

since you hardcoded those 2 keys since the beginning when otg mode feature was created there's no version of scrcpy i can revert back to achieve what i want, which left me with a remaining option is to find a way to configure my specific environment so it will only capture Super key, would that be possible now?

@rom1v
Copy link
Collaborator Author

rom1v commented Mar 6, 2024

since you hardcoded those 2 keys since the beginning when otg mode feature was created there's no version of scrcpy i can revert back to achieve what i want

This is open source. Clone the project, change the line to adapt, rebuild 😉

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants